This PR made the following changes to the salt library for running latest whisper finetuning :

• during dataset loading:
  • add support for google cloud buckets.
  • add supports to limit number of examples per dataset.
  • skips the complicated source-targe matching logics for ASR model with skip_matching_asr argument.
  • iterate validation datasets also with multi-threads while keeping the same order.
• update SALT_LANGUAGE_TOKENS_WHISPER in constants.py with 51 African languages for new whisper-salt ASR model training.
• optimize the speech of multilingual_eval_fn in metrics.py by skipping unnecessary heavy CPU audio decoding process.
• fixed a bug in augment_audio_noise function in preprocessing.py that makes the output audio to be zero-size.

The whisper finetuning/training scripts and configs has been moved sunbird-speech repo under speech-to-text/whisper directory.

[Depreciated]

This PR adds the support of google cloud buckets for the whisper training pipeline, and made several other changes:

• load the parquets datasets from gcs:// path with datasets.load_dataset and cast the audio column to datasets.Audio format.
• create a setup shell script that intalls the dependencies and configure the google cloud credentials.
• load modules such as salt.datasets from the current repo instead of https://github.com/jqug/salt.git
• move the yaml config from training notebook to a separate file
• fix several training errors with following changes:
  • Use BF16 instead of FP16 for training
  • set gradient_checkpointing=False
  • add torch_dtype=torch.float32 when loading the model weights
  • updates model.generation_config based on requirements from the new version.

Overfit experiment

An overfit experiment with just 100 examples was done to verify the changes:
MLflow run1 with evaluation metrics: https://mlflow-sunbird-ce0ecfc14244.herokuapp.com/#/experiments/0/runs/2d488acdc39146e9af9da07c00128d49/model-metrics
MLfLow run2 with GPU utilization: https://mlflow.sunbird.ai/#/experiments/0/runs/811bbdf051f44597bd90c3376cfc9309/system-metrics

TODO

• we need to update the salt.constants.SALT_LANGUAGE_TOKENS_WHISPER to support new languages. Currently we only have the following:

SALT_LANGUAGE_TOKENS_WHISPER = {
    # Exact/close mapping
    'eng': 50259,
    'swa': 50318,
    # Overwrite unused language tokens
    'ach': 50357,
    'lgg': 50356,
    'lug': 50355,
    'nyn': 50354,
    'teo': 50353,
    'xog': 50352,
    'ttj': 50351,
    'kin': 50350,
    'myx': 50349,
    'kik': 50348,
}

• currently each evaluation step takes 3-4 mins. I'm not sure whether it is expected